Skip to content

Standalone force-catch-up for Marten under Wolverine-managed distribution (GH-3697) - #3735

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3697/marten-force-catchup
Jul 30, 2026
Merged

Standalone force-catch-up for Marten under Wolverine-managed distribution (GH-3697)#3735
jeremydmiller merged 1 commit into
mainfrom
gh-3697/marten-force-catchup

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3697.

We already had most of this

Under UseWolverineManagedEventSubscriptionDistribution the Marten store runs in DaemonMode.ExternallyManaged, and since marten#4904 IHost.ForceAllMartenDaemonActivityToCatchUpAsync() is deliberately a passive read-only wait in that mode — it punts active catch-up to "the external coordinator", i.e. Wolverine.

Wolverine already implements that coordinator path — catchUpThroughCoordinatorAsync, added in GH-3349 — but only exposes it as a TrackActivity() stage, PauseThenCatchUpOnMartenDaemonActivity(). A suite that isn't driving its work through a tracked session had nothing to call, which is exactly the gap reported.

What's added

The standalone entry point, on IHost and IServiceProvider, plus the <T> ancillary-store variants:

// Drive every projection and subscription to the current high water mark, leave the daemons running
await host.PauseThenCatchUpOnMartenDaemonActivityAsync();

// ...or leave them paused, so nothing races your assertions
await host.PauseThenCatchUpOnMartenDaemonActivityAsync(CatchUpMode.AndDoNothing);

// Ancillary stores take the store type
await host.PauseThenCatchUpOnMartenDaemonActivityAsync<ILetterStore>();

Against the three asks in the issue:

  1. "suspends agent assignment cluster-wide so the forced catch-up really is the only writer" — approached from the other side. Rather than stopping the shards' own agents and becoming a competing writer, the helper resumes the agents that already own those shards and waits for non-stale data. There is only ever one writer, so "sole writer" isn't best-effort here, it's structural.
  2. "drives every locally reachable daemon, including per-(database, tenant) shards"WaitForNonStaleProjectionDataAsync covers every database in the store.
  3. "restores the previous assignment state afterwards"CatchUpMode, applied in a finally so a timeout still restores it.

On the two exceptions in the report

ProgressionProgressOutOfOrderException and 23505 ... pk_mt_event_progression are not races the supported path swallows — they are the signature of the workaround's daemon.StopAllAsync() + daemon.CatchUpAsync() under a paused coordinator. CatchUpAsync is the second writer. This helper never calls it, so neither error has anything to arise from. Nothing needs to be retried or swallowed.

Tests

MartenTests/TestHelpers/standalone_catch_up_under_wolverine_distribution.cs, all green locally:

  • main store under UseWolverineManagedEventSubscriptionDistribution, no tracked session;
  • ancillary store;
  • AndDoNothing leaves the daemons paused, and a second pass still catches up events appended while they were paused — the exact state the hand-rolled version was fighting.

Full wolverine.slnx Release build clean.

Docs

New "Forcing projection catch-up outside a tracked session" section in the testing guide, with an explicit warning against the pause-then-CatchUpAsync pattern and why it produces those two errors.

🤖 Generated with Claude Code

…ibution (GH-3697)

Under `UseWolverineManagedEventSubscriptionDistribution` the Marten store runs
in `DaemonMode.ExternallyManaged`. Since marten#4904,
`IHost.ForceAllMartenDaemonActivityToCatchUpAsync()` is deliberately a passive
read-only wait in that mode -- it no longer builds projections on demand, and
it explicitly punts active catch-up to "the external coordinator", i.e.
Wolverine.

Wolverine already implements that path, in `catchUpThroughCoordinatorAsync`
(GH-3349), but only exposes it as a `TrackActivity()` stage
(`PauseThenCatchUpOnMartenDaemonActivity`). A suite that isn't driving its work
through a tracked session had nothing to call, so it reimplemented the
quiesce-then-catch-up dance by hand -- and the obvious hand-rolled version
races.

Adds the standalone entry point on `IHost` and `IServiceProvider`, plus the
`<T>` ancillary-store variants:

    await host.PauseThenCatchUpOnMartenDaemonActivityAsync();
    await host.PauseThenCatchUpOnMartenDaemonActivityAsync(CatchUpMode.AndDoNothing);
    await host.PauseThenCatchUpOnMartenDaemonActivityAsync<ILetterStore>();

Same algorithm the tracked-session stage runs, minus the message-tracking
bookkeeping. The important part is what it does NOT do: it never calls
`IProjectionDaemon.CatchUpAsync`. Calling that while the coordinator owns the
shards is what adds a *second* writer of each progression row and produces the
`ProgressionProgressOutOfOrderException` and `23505 pk_mt_event_progression`
duplicate-key errors reported on the hand-rolled workaround. Resuming the
agents that already own those shards and waiting for non-stale data means
there is only ever one writer, so neither race exists to be retried around and
swallowed.

`CatchUpMode` governs the state afterwards, in a `finally`, so a timeout still
restores it.

Tests cover the main store, an ancillary store, and that `AndDoNothing` leaves
the daemons paused and a second pass still catches up events appended while
they were paused -- the exact state the hand-rolled version was fighting.

Docs: a "Forcing projection catch-up outside a tracked session" section in the
testing guide, including a warning against the racing pattern.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 76c8a78 into main Jul 30, 2026
32 checks passed
@jeremydmiller jeremydmiller mentioned this pull request Jul 30, 2026
This was referenced Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant